Catheterscope 3D guidance and interface system

ABSTRACT

Visual-assisted guidance of an ultra-thin flexible endoscope to a predetermined region of interest within a lung during a bronchoscopy procedure. The region may be an opacity-identified by non-invasive imaging methods, such as high-resolution computed tomography (HRCT) or as a malignant lung mass that was diagnosed in a previous examination. An embedded position sensor on the flexible endoscope indicates the position of the distal tip of the probe in a Cartesian coordinate system during the procedure. A visual display is continually updated, showing the present position and orientation of the marker in a 3-D graphical airway model generated from image reconstruction. The visual display also includes windows depicting a virtual fly-through perspective and real-time video images acquired at the head of the endoscope, which can be stored as data, with an audio or textual account.

RELATED APPLICATIONS

This application is a continuation application of Ser. No. 11/342,074filed Jan. 27, 2006, now U.S. Pat. No. 8,382,662, issued on Feb. 26,2013, which is a divisional application based on prior copendingapplication Ser. No. 11/009,699 filed Dec. 10, 2004, now U.S. Pat. No.7,901,348 issued on Mar. 8, 2011, which itself is based on provisionalapplication Ser. No. 60/529,077 filed Dec. 12, 2003, the benefit of thefiling dates of which are hereby claimed under 35 U.S.C. §§120 and119(e).

FIELD OF THE INVENTION

This invention generally relates to a method and apparatus for providingthree-dimensional (3-D) guidance to a catheterscope or flexibleendoscope that is being advanced through a branching lumen in apatient's body, and more specifically, to the use of a 3-D model inconnection with a sensor, for determining the disposition andorientation of a flexible endoscope, for visually guiding the flexibleendoscope through a series of branching lumens in a patient's body.

BACKGROUND OF THE INVENTION

On high-resolution computed tomography (HRCT) scans, potentiallycancerous masses appear as radio-opaque nodules. “Screening” HRCT scansare now offered commercially to target patients at high risk for lungcancer. Modern high-resolution systems are now able to identify manysmall, potentially cancerous lesions not previously visible in suchscans. However, these lesions pose a difficult diagnostic problem;because they are small and peripherally disposed in the lung, they aredifficult to reach in order to take a tissue sample. To diagnose cancer,a tissue or cellular sample at a suspect site is often acquired eitherby transthoracic needle aspiration or during a bronchoscopy procedure.In the bronchoscopic procedure, small forceps, a fine needle, or acytological brush are advanced through a biopsy channel of a flexiblescope and inserted into the major lobes of the affected lung to reach asuspect site that is disposed near a relatively large airway. Currentbronchoscopes are only able to fit within the relative large branches ofthe bronchial system.

Transthoracic needle aspiration is very invasive and is typicallyreserved for peripheral lung nodules suspected of being cancerous.However, transthoracic needle aspiration can compromise patient healthand produce infections. To minimize damage to surrounding tissue andorgans, clinicians rely heavily on fluoroscopic C-arms and HRCT to guidethe needle to the location of the identified suspect tissue mass. Manyprocedures are performed while the patient is within the CT suite toenable a biopsy needle to be inserted, steered, and continuallyre-imaged with a fluoroscope in incremental steps to verify its positionrelative to a suspected tumor. Unfortunately, transthoracic needleaspiration can compromise patient health and requires prolonged use ofimaging systems that impose a substantial financial expense on thepatient, and a high time cost on the hospital. Thus, bronchoscopy is amore preferred method of extracting lung tissue for biopsy purposes thanis transthoracic needle aspiration.

Bronchoscopy involves the insertion of a fiber-optic bundle into thetrachea and central airways in a patient's lung. Navigation of theairway relies on the clinician's ability to direct the scope head intothe appropriate region. Histological biopsy samples can be obtainedusing forceps, a transbronchial needle aspiration, with brush cytology,or by bronchial lavage. Though still invasive, this method is much saferand is not considered to be traumatizing to the patient, in contrast tothe transthoracic needle aspiration method. Despite this benefit, thelarge diameter of commercially available bronchoscopes restricts theirentrance into small airways where nodules are commonly found, thusrequiring the clinician to either steer the forceps/needle/brush blindlyor use fluoroscopy to navigate to these regions throughout the lung inhopes that a representative specimen is obtained from the site actuallyidentified as potentially cancerous.

At present, fluoroscopic C-arms are commonly used in bronchoscopy suitesto assist clinicians in navigating the airways by projecting orthogonalviews of the thoracic cavity in real-time. Unfortunately, drawbacks ofthis method are: (1) maneuvering a catheter in two separate planes isperceptually awkward; (2) images provided by a conventional bronchoscopeare unclear or “fuzzy” and it is relatively difficult to see anatomicaldetail; (3) it is cumbersome to continually adjust the C-arm; (4) theradiation load associated with continued fluoroscopy is detrimental tothe health of both the patient and the physician. Also, positioncoordinates of the bronchoscope cannot be measured or calculated with afluoroscope, precluding its integration into any graphic interface andmaking it difficult to keep a historical record of the path followed bythe bronchoscope, should such a record be needed during follow upexaminations.

An ideal strategy for detection of suspect nodules in the lung wouldinvolve a minimally invasive biopsy of the potentially cancerous tissuemass, optical imaging of the epithelial layer, and real-time archivingof examination findings—without the need for expensive, radiation-basedimaging systems. Ideally, it should be possible to visually guide abronchoscope through very small airways, while maintaining a repeatablereference to the location of the bronchoscope in the airways. Inaddition, it would be desirable to produce data that show the pathsfollowed and the regions of the airways that were visited during thebronchoscopy to enable a physician to easily revisit a specific nodulelocation at a later time, with minimal time required to retrace thebranching path that must be followed to reach that location. The datarecorded and stored during such a guided bronchoscopy would enable aphysician to contest a charge that the physician failed to takeappropriate steps to detect a malignant tissue site in the lungs, shoulda charge of malpractice arise.

The benefits of visually guiding a device through a lumen in a patient'sbody are not limited to bronchoscopes or to diagnostic studies of thelung. Clearly, it would also be desirable to guide a flexible endoscopethrough other body lumens, for example, through a gastric lumen so as toavoid entering the pancreatic duct in a patient when advancing anendoscopic probe beyond the pyloric valve of the stomach. A flexibleendoscope might also be more efficiently advanced through thecardiovascular system of a patient if it were possible to visualize theanatomical disposition of the endoscope with reference to its locationin a 3-D model of the system as determined with a position sensor.Additionally, an ultra-thin flexible endoscope and position sensor couldalso be used for navigating through the urethra for an image-guidedbiopsy of the prostate, which is not possible with conventionalendoscopes. Other applications of this technology will undoubtedly bemore apparent when its capabilities are fully realized by the medicalprofession.

SUMMARY OF THE INVENTION

The present invention enables a visually-assisted guidance of anultra-thin flexible endoscope to a predetermined region of interestthrough a lumen in a patient's body. This region may be an opacityidentified by non-invasive imaging methods, such as HRCT or a malignantlung mass diagnosed in a previous examination. A position sensor on theendoscope produces a signal indicating the position (and orientation) ofthe distal tip of the endoscope in a Cartesian coordinate system duringthe procedure. A visual display is continually updated, showing thepresent position and orientation of the marker in a 3-D graphicalsurface model of the airways is generated through segmentation ofmedical images. The visual display also includes windows depicting avirtual fly-through perspective and real-time video images acquired atthe distal tip of the endoscope, which can be stored as data.Optionally, an audio or textual account can be included with data andstored for subsequent review.

In addition to the surface model, an airway tree model is constructed tooutline the hierarchy and connectivity of bifurcations originating atthe main carina and extending down the bronchial tree. Within thisairway tree model, branch points are represented by their 3-D position,a generation index that specifies how many levels separate the node fromthe main carina, the branching angles of the two subtending vessels, anda definition of the centerlines that connect two linked branch points.Based on the known position of potentially cancerous lesions on a HRCTscan, a number of nearby biopsy points are selected for inspection. Fromthese points, a series of courses are automatically plotted in the modeltree to effectively steer the physician at each branching whilerecording and graphically indicating regions previously inspected duringthe procedure to prevent over-sampling of the same region as well asensuring a comprehensive probing of potentially affected areas.

The identification of bifurcations by the clinician on video imagesserves to confirm that the measured real-time position of the scope headaligns with its inferred position on the visual 3-D model andcontinually recalibrates the two to reduce accumulating errors.Recalibration involves a point to point re-registration of the lungmodel bifurcation location to the known position of the tracker in 3-Dspace. As a result, the position of the scope within a set of tightlybranched vessels can be deduced using knowledge of the decision historywhere a decision constitutes the protrusion of the scope into one of twoor more daughter vessels at any given carina.

Accordingly, one aspect of the present invention is directed to a systemfor visually guiding a flexible endoscope through linked passages withina body. The system includes a signal source that emits a referencesignal useful for spatially tracking the progress of the flexibleendoscope through the linked passages, and a sensor that produces anoutput signal indicating a 3-D disposition of a distal end of theflexible endoscope using the reference signal of the signal source. Thesignal source can either be included adjacent to the distal end of theflexible endoscope or can be disposed external to a patient's body. Ifthe signal source is external, the sensor is disposed adjacent to thedistal end of the flexible endoscope, while if the signal source ismounted on the flexible endoscope, the sensor is disposed externally. Auser interface is included for displaying a view from within the linkedpassages as the flexible endoscope is advanced therethrough. Thedisplayed image enables the flexible endoscope to be visually guided andtracked along a path being followed through the linked passages.

The system preferably employs a 3-D model of the linked passages that isused to assist in determining at least an initial path along which toadvance the flexible endoscope. The 3-D model is preferably derived froman image of a portion of a body in which the linked passages aredisposed, for example using the images produced by a CT or MRI scan.This model provides an indication of an orientation and a disposition ofthe linked passages relative to a specific anatomical feature, such asthe carina in the lungs, a bone structure, or other known easilyidentified feature.

While it is possible to use long wavelength infrared signals thatpenetrate tissue to track the flexible endoscope, the signal sourcepreferably comprises an electromagnetic field source. In this case, thesensor preferably comprises at least three coils configured to sense atleast a 3-D position of a distal portion of the flexible endoscoperelative to the electromagnetic field source. The sensor comprises atleast two coils configured to sense both a 3-D position and anorientation of the distal portion of the flexible endoscope in sixdegrees of freedom. Only one coil is needed if it is desired to trackposition and orientation of the flexible endoscope in five degrees offreedom, as it is advanced through the linked passages.

Optionally, the system also includes a body function sensor configuredto be disposed adjacent to either the sternum, the thorax, and the mouthof a patient, to monitor one or more of breathing, chest orientation,and the deformation of the lungs of a patient. This sensor produces datathat can be employed for updating the 3-D model in regard to changescaused by a body function. Also, the signal from this sensor is usefulfor determining the disposition of the distal end of the flexibleendoscope in regard to the at least one body function, or for gating theflexible endoscope to synchronize with the body function when sensingthe position/orientation of the flexible endoscope, thereby minimizingthe effect of the body function.

In a preferred embodiment, the flexible endoscope includes a vibratingoptical fiber that scans a region adjacent to the distal end of theflexible endoscope, producing a signal that is used to drive thedisplay. This configuration is very compact, yet produces much higherresolution images than is possible with prior art optical fiberendoscopes.

Other aspects of this invention are directed to a flexible endoscope,generally as described above, and to a method for guiding a flexibleendoscope through linked passages to either perform a diagnosticprocedure or reach a desired location in a patient's body, whiletracking the disposition of a distal end of the flexible endoscope. Themethod includes the step of using the flexible endoscope for scanning aregion of the linked passages that is adjacent to the distal end of theflexible endoscope, to display an image of the region. An externaltracking signal is employed and serves as a reference relative to thelinked passages within the patient's body. The method thus includes thestep of tracking the disposition of the distal end of the flexibleendoscope as it is advanced through the linked passages, using an outputsignal produced in response to sensing the external tracking signaladjacent to the distal end of the flexible endoscope. The distal end ofthe flexible endoscope is then advanced through the linked passagesalong a path determined at least in part by the image of the region andby the output signal, to reach the desired location in the patient'sbody. Other steps of the method are generally consistent with thefunction of the system discussed above.

BRIEF DESCRIPTION OF THE DRAWING FIGURES

The foregoing aspects and many of the attendant advantages of thisinvention will become more readily appreciated as the same becomesbetter understood by reference to the following detailed description,when taken in conjunction with the accompanying drawings, wherein:

FIGS. 1A and 1B are, respectively, a functional block diagram of theflexible endoscope system, and a cross-section of a rigid, distal end ofa flexible endoscope that includes optical fibers for conveying lightback to detectors (not shown) at a proximal end of the flexibleendoscope, where the light was emitted by the scanning optical fiber andreflected by an adjacent surface back into the optical fibers;

FIGS. 1C and 1D are, respectively, a perspective schematic view and anend view of a distal portion of a flexible endoscope that includes Red,Green, Blue (RGB) light sensors for sensing light that has been emittedby a scanning optical fiber and which has been reflected from anadjacent surface;

FIG. 1E is a schematic cross-sectional view of the distal end of aflexible endoscope that includes a triangular mirror responds toparallel polarized or scattered light, and to perpendicular polarized orfluorescent light received from surfaces adjacent to opposite sides ofthe distal end of the flexible endoscope;

FIG. 1F is a schematic cross-sectional view of the distal end of anotherembodiment of the flexible endoscope that includes a biopsy channel anda biopsy needle within the channel that can be selectively advanced intotissue to take a sample;

FIGS. 1G and 1H are, respectively, a schematic transversecross-sectional view near the proximal end and a schematic transversecross-sectional view of the rigid portion (near the distal end) of theflexible endoscope of FIG. 1F;

FIG. 1I is a schematic view of another embodiment that includes acytological brush for taking a biopsy sample (for purposes ofsimplifying the image, much of the detail is omitted in this Figure);

FIG. 2A is a 3-D CT-based surface model and overlying tree model of thebronchial airways, indicating identified nodes and centerlines detectedfrom the skeletonized lung volume, with an origin (0,0,0) set at a maincarina;

FIG. 2B is an example of a branching node hierarchy specified by i−j,where i is the generation, and j is its index within a model coordinateframe (MCF);

FIG. 2C is a schematic diagram in which a bold arrow points to an innerpoint of a confluence of two passages into a passage that leads to themain carina;

FIG. 2D is a schematic of an airway surface model with an overlay of anairway tree model, showing how a detected branch point is projected toan inner point of confluence of the surface model for manualreregistration at automatically selected points;

FIG. 3A is a schematic view of an airway tree model generated from aseries of transverse HRCT scan slices;

FIG. 3B is a schematic view of a calibration of a position sensor usedon a flexible endoscope in accord with the present invention, byselecting the main carina as the origin (i.e., a reference point), andusing a magnetic field transmitter to measure position and orientationrelative to the reference point;

FIG. 4A is a constructed airway surface model, showing selected pointstargeted for a biopsy, and an automated course planning through the lungpassages (where the generation index of each branch point is indicated);

FIGS. 4B, 4C, and 4D are, respectively, an exemplary user interfacedepicting a real-time image of a bronchial passage produced by theflexible endoscope, a virtual analog of the same perspective constructedfrom HRCT scans, and a static 3-D airway surface model in which apresent course or path is plotted along the centerlines of the airwaytree model passages, along with a current position of the flexibleendoscope;

FIG. 5A is a schematic diagram showing how the 3-D model is calibratedto adjust for an accumulated error by comparison of the model with theactual position of the distal end of the flexible endoscope;

FIG. 5B illustrates how color coding is employed to mark airways thathave already been inspected, e.g., with a red marker, and an intendedcourse of navigation, with a green marker;

FIG. 5C is an image illustrating segmentation of low light intensityregions of subtending vessels and an axis showing 2-D deflection angles;

FIG. 5D illustrates how a 3-D model is extended by mapping a position ofthe flexible endoscope on the model coordinates at open vesselterminations;

FIG. 6 illustrates an example showing how motion vector correction isapplied by assuming forward movement of the flexible endoscope, toaccount for tidal breathing in the small lung passages;

FIG. 7A illustrates a system in accord with the present invention inwhich an external electromagnetic field generator is employed as asignal source, for determining a position of an internal sensor disposedadjacent to the distal end of the flexible endoscope;

FIG. 7B illustrates a system in accord with the present invention inwhich an internal electromagnetic field source adjacent to a distal endof the flexible endoscope is employed with an external sensor, fordetermining a position of the distal end of the flexible endoscope;

FIG. 7C illustrates a system in accord with the present invention inwhich a sensor is disposed around the thorax of a patient to monitorbreathing, so that errors in the position of the distal end of theflexible endoscope can be compensated for the effects of breathing andother physiological body functions;

FIG. 7D illustrates a system in accord with the present invention inwhich a an imaging device is shown imaging a plurality of markersapplied to the chest of a patient to monitor breathing and movement, sothat errors in the position of the distal end of the flexible endoscopecan be compensated;

FIG. 7E illustrates how a plurality of motion sensors affixed to thethorax of a patient can be used to provide bodily function signals thatcan be employed to compensate for movement of the patient's lungs andbronchial passage;

FIG. 8A is a flowchart showing the functions steps employed in thepresent invention;

FIG. 8B is a functional block diagram illustrating the different methodsthat can be employed to update the 3-D model;

FIG. 8C is a flow chart illustrating details of the step for using theshortest or most direct path to navigate when moving beyond the extentof the 3-D model;

FIG. 8D is a flow chart illustrating details of the step forautomatically tracking position in the MCF and re-registering to the ACFusing the endoscopic images at a bifurcation of the passages;

FIG. 8E is a flow chart illustrating details of the step for adaptivenon-rigid registration of the MCF and ACF;

FIG. 8F is a flow chart illustrating details of the step for adaptiverigid registration of the MCF and ACF;

FIG. 9A is a schematic diagram of a dipole electromagnetic fieldtransmitter and sensor capable of sensing six degrees of freedom of thedistal end of the flexible endoscope;

FIG. 9B is a schematic diagram of a sensor with two coils that iscapable of sensing six degrees of freedom of the distal end of theflexible endoscope; and

FIG. 9C is a schematic diagram of a sensor with a single coil that iscapable of sensing five degrees of freedom of the distal end of theflexible endoscope.

DESCRIPTION OF PREFERRED EMBODIMENTS

Ultra-Thin Flexible Endoscopes

Ultra-thin flexible endoscopes have been devised for diagnostic imagingof small lumens with diameters similar to regions in the peripheral lung(i.e., with diameters less than 3 mm). A flexible endoscope of thiscapability has been achieved mainly by reducing the number of fiberscontained within the imaging bundle. The disadvantage of this solutionis that in removing fibers, the resolution and field of view are greatlydiminished. Space restrictions preclude incorporation of a biopsychannel into the design. Although quite thinner than their predecessors,these flexible endoscopes also remain fairly expensive and often breakat small bending radii. The poor imaging ability, fragility, expense,and inability to obtain tissue specimens deters the use of this type ofbronchoscope for examination of masses in the peripheral lung.

The development of a 2 mm diameter flexible endoscope in accord with thepresent invention enables clinicians to access peripheral airways of thelung to biopsy nodules identified on HRCT images. Embedded within apreferred embodiment of the flexible endoscope is a single-mode opticalscanning fiber offering a wide angle field of view, as well as a wiredposition-sensor that is used to locate the distal tip of the catheter in5 or 6 degrees of freedom, with an accuracy of around 1 mm. While thistechnology permits extension of this device to more distant regions ofthe lung, it becomes imperative that some guidance system beincorporated to assist in navigating this flexible endoscope throughpathways potentially leading to the region of interest.

Motivation for Position Tracking

The lung is a series of bifurcating airways ranging in diameter from 18mm to 0.25 mm. Conventional fiber bundle bronchoscopes designedprimarily for larger lumens are capable of reaching 5^(th) generationbronchioles amounting to a total of 2⁵ or 32 separate airways. Theultra-thin flexible endoscope is able to extend to the 8^(th) generationcomprising a total of 2⁸ or 256 distinct airways. Though navigation ofthese devices typically relies on an optical view from the head of thescope, this does not necessarily aid in directing the catheter through asystem where there is extensive branching. The exponential increase incomplexity underscores the need for some means of visually tracking theposition of this endoscope on a HRCT generated 3-D model so that it mayeffectively guide the physician at each branch point as well asrecording the regions that have been inspected. By implementing aguidance system within the ultra-thin flexible endoscope mentionedabove, procedures can be performed easily within general examinationrooms without need for C-arm fluoroscopy units, CT or MRI scanners, andthe highly specialized environments that occupy these systems.

Position Tracking

A position tracker serves as a 3-D guidance system by providing absolutemeasurements of the tracker's position with reference to some externalsource. The three orthogonal coils that constitute the sensor generatevoltages when placed within a magnetic field. These signals can bemeasured and digitized to determine position within six degrees offreedom. Magnetic field position sensors have diminished in size overthe years, promoting their use in many medical applications wherevirtual navigation has eliminated the need for radiation-based imagingmethods. Miniature sensors have been embedded in electrophysiologycatheters used for cardiac navigation and arrhythmia treatment (CARTOXP™, available from Biosense Webster of Diamond Bar, Calif.). Similarsystems use electric potential to measure position (Localisa™ fromMedtronic of Minneapolis, Minn.; and EnSite NavX™ from EndocardialSolutions of St. Paul, Minn.). Overall, these systems are used formodeling voltage potentials in the heart and are not concerned withtraversing tortuous branches or with path finding. Because thesecatheters contain electrodes rather than imaging optical fibers, theirdesign restraints are far less stringent. However, the present inventionhas established the use of electromagnetic and electric potentialtracking systems as a reliable means for measuring position in vivo,within a clinical setting. As well, the system components used in thepresent invention are far more accessible and inexpensive thanfluoroscopy units, CT scanners, MRI apparatus, or other large imagingsystems that are often immobile or at least difficult to move, occupy alarge space, and require specialized suites and technicians, and requireconsiderable maintenance to keep operational.

Unfortunately, position itself is not sufficient for navigating acatheter through a series of diverging passages such as the bronchialairways of the lung. Sources of error, such as inaccurate readings, poormorphological correlation with the static 3-D model, patient movement,breathing, deformation induced by the catheter, or other perturbationsmay be too great to ascertain the location of the catheter tip withinone of many small bronchial passageways.

One aspect of the present invention is directed to a system forsupervised steering of a flexible endoscope through the bronchial tree.The system relies on the visual identification of branch points by aphysician, to continually recalibrate the current position of theflexible endoscope to the corresponding branch point on the static 3-Dmodel. This methodology uses measurements of absolute position relativeto a sensor, in order to generate positional data comprising a device“history” that simplifies the position of the flexible endoscope to aseries of choices made along a binary decision tree in which thedecisions determine which branch to take with the flexible endoscope ateach junction of the bronchial tree. Incorporated within this frameworkare several approaches for reducing the measurement error, given thenumber of perturbations to the system. A simplified decision algorithmis also described for piloting the flexible endoscope to regions withinsmall airways that are missing in the reconstructed model, due to thelimited image resolution and partial volume effects associated with CTscanning.

However, it must be repeatedly emphasized that the present invention isnot limited only to traversing bronchial passages, since the presentinvention can be employed for traversing other types of linked passageswithin a patient's body, besides those in the bronchial system. The sameapproach is used in almost any such application of the presentinvention, for guiding the flexible endoscope through the linkedpassages. Accordingly, although the focus of this disclosure is directedto the use of the system for navigating a flexible endoscope throughbronchial passages, the discussion presented herein should be understoodto apply to many other applications of this invention.

Details of the Flexible Endoscope System

An exemplary flexible endoscope system 20 illustrated in FIG. 1A. In asystem, a flexible endoscope 24 is inserted through a multi-functionendoscopic catheter 22, which facilitates accessing passages of interestwithin a patient's body, such as the bronchial passages within thelungs. Flexible endoscope 24 includes a relatively rigid portion 26,which is disposed at its distal end; details of several differentembodiments of the flexible endoscope are discussed below, particularlyin regard to the components used for scanning and imaging tissue that isadjacent to the distal end of the flexible endoscope. The proximal endof the flexible endoscope includes a rotational control 28 and alongitudinal control 30, which respectively rotate and move the flexibleendoscope longitudinally relative to catheter 22, providing manualcontrol for one-axis bending and twisting. Various electrical leadsand/or optical fibers (not separately shown) extend through a branch arm32 to a junction box 34.

Light for scanning tissue at the distal end of the flexible endoscopecan be provided either by a high power laser 36 through an optical fiber36 a, or through optical fibers 42 by individual red, green, and bluelasers 38 a, 38 b, and 38 c, respectively, each of which can beindividually modulated and the colored light combined into a singleoptical fiber 42, using an optical fiber combiner 40. A signalcorresponding to light that is reflected from tissue adjacent to thedistal end of flexible endoscope 24 can either be detected with sensorsdisposed adjacent to the distal end, or can be conveyed through moreoptical fibers that extend back to junction box 34.

This signal is processed by several components, including a component 44that is connected a junction box 34 through leads 46 calculates imageenhancement and provides stereo imaging of the scanned region. Alsoprovided are electrical sources and control electronics 48 for opticalfiber scanning and data sampling, which are coupled to junction box 34through leads 50. A sensor (not shown in this figure) provides signalsthrough leads 54 that enable electromagnetic position tracking of thedistal end of the flexible endoscope in vivo, indicating its position(and optionally, its orientation) with up to five to six degrees offreedom, as indicated in a block 52.

Leads 58 connect junction box 34 with an interactive computerworkstation and monitor 56, which has a keyboard and/or other user inputdevice 60. Optionally, the interactive computer workstation is connectedto a high resolution color monitor 62, which can display very detailedvideo images of the passage through which the flexible endoscope isbeing advanced.

Embodiments of Flexible Endoscope

In the best scenario, the implementation and operation of thisnavigation scheme relies on several components, including a static 3-Dgraphical model of the airspaces, a forward viewing bronchoscope orflexible endoscope with a digital video output, a monitor, a processor,and appropriate software to carry out the processing for implementingthe steps of the navigation scheme in its various embodiments. Thisinvention was designed particularly to employ a novel ultra-thinflexible endoscope such as shown in FIGS. 1B, 1C and 1D, and 1E. Each ofthese embodiments includes a housing 80 (FIG. 1B), 80′ (FIGS. 1C and1D), and 102 (FIG. 1E) having an outer diameter of about 2 mm or less.This small diameter permits extension of the device into previouslyinaccessible regions of the lung or other body passages where there isextensive branching, requiring a tracking method to determine thelocation of the flexible endoscope in the passage. However, thisinvention can also be very useful when the flexible endoscope isadvanced through any passage in which branching occurs. A wired positionsensor 84, which disposed adjacent to the distal end of a flexibleendoscope 24, as shown in FIG. 1B, measures the orientation and positionwithin five or six degrees of freedom in connection with a system 20, asillustrated in FIG. 1A.

Flexible endoscope 24, which is shown in FIG. 1B, includes a positionsensor 84 with three orthogonal coils (not separately shown) thatproduce signals indicative of the position and orientation of rigidportion 26 of the flexible endoscope relative to an electromagneticfield source (not shown in this Figure), which is external to thepatient's body in this embodiment. The signals produced by the sensorare conveyed through electrical leads 86 to the interactive computerworkstation and monitor, which processes the signal to determine theabsolute position and orientation of the distal end of flexibleendoscope 24 relative to the electromagnetic field source. The signalindicative of position and orientation thus enables the user todetermine where the distal end of the flexible endoscope is located inregard to a 3-D coordinate system based upon the location of theelectromagnet field source.

Optionally, a balloon 88 can be inflated to stabilize the flexibleendoscope within a passage of the patient's body. When balloon 88 isinflated, the flexible endoscope is held in place, which can resistmovement of the distal end when a biopsy needle is inserted andretracted or can stabilize the flexible endoscope so that the movementof air through the bronchial passages does not displace it.

A key advantage of the present invention is that the user can view thepath being followed by the flexible endoscope through linked passages inthe patient's body, by displaying the video images produced by ascanning optical fiber 72, which is disposed in the rigid portion at thedistal end of the endoscope. The scanning optical fiber in thisembodiment is driven to move in a spiral scan by a two-axispiezoelectric driver 70. Light conveyed through a single mode opticalfiber 74 to the scanning optical fiber from an external laser or otherlight source and is directed first through a lens 76 and then through alens 78. These two lenses focus the light emitted by the scanningoptical fiber onto an adjacent surface. Light reflected from theadjacent surface passes back through lenses and is conveyed multimodeoptical fibers 82 a and 82 b, which respectively include lenses 82 a′and 82 b′, back to the proximal end of the flexible endoscope, wherelight detectors (not shown) are provided, or alternatively, thedetectors can be included in the rigid portion of the endoscope, so thatelectrical signals from the light detectors are conveyed throughelectrical leads (running generally like multimode optical fibers 82 aand 82 b, but much smaller in diameter) to the processing hardware thatis external to the patient, enabling a video image of the region throughwhich the flexible endoscope is moving to be viewed by the user.

FIGS. 1C and 1D illustrate flexible endoscope 24′, which is much likeflexible endoscope 24, except that a rigid portion 26′ includes redlight detectors 92 a, green light detectors 92 b, and blue lightdetectors 92 c spaced closely around scanning optical fiber 72 at theend of a support cylinder 94, and more radially outwardly disposed redlight detectors 96 a, green light detectors 96 b, and blue lightdetectors 96 c that are set back from lenses 76 and 78. These sensors,which are each sensitive to a specific spectral range (e.g., a specificcolor of light), can comprise a photodiode material with an appropriatefilter cap. The redundant light detectors of each color are provided sothat ambient light levels can be recorded and specular reflections canbe excluded in the image that is detected in response to light reflectedfrom adjacent tissue by excluding the detector of each color that hasthe substantially higher level output, which is the light detector thatis receiving specular reflection from the tissue rather than diffuselyreflected or scattered light.

Also shown in FIGS. 1B and 1C are touch sensors 98 that are spaced apartaround the periphery of lens 76. Touch sensors 98 comprises capacitiveor piezoelectric sensors that respond to contact with tissue, e.g., atthe junction of two passages that branch off of the passage in which theflexible endoscope is being advanced. The signal produced by touchsensors 98 is thus useful in determining when the distal end of theflexible endoscope has contacted the junction, which serves as referencefor tracking the flexible endoscope and providing registration betweenthe MCF and ADF.

In FIG. 1E, a flexible endoscope 100 is illustrated that is capable ofimaging adjacent regions to the sides of the flexible endoscope, and asshown in the example, using different light modalities. This embodimentthus transmits scanning light transversely of the longitudinal axis ofdistal tip sheath or housing 102 and receive light reflected back fromthe adjacent tissue to the sides of the distal end of the flexibleendoscope. Specifically, in addition to single mode optical fiber 74,which is used to convey light to illuminate the regions on oppositesides of the housing as scanning optical fiber 72 moves in a spiralmode, flexible endoscope 100 also includes three collection opticalfibers 106 that convey either parallel polarized light or scatteredlight collected from adjacent tissue disposed to the side of thehousing, and three collection optical fibers 108 that convey eitherperpendicular polarized light or fluorescent light from adjacent tissueon the opposite side of the housing. The proximal ends of these opticalfibers are illustrated at a proximal end 104 of the flexible endoscope,where appropriate filters (not shown) might be applied, as appropriate.The distal ends of the collection optical fibers are secured in a mount110 to receive light that is reflected through lenses 112 a and 112 b,having passed through lens 116 a or lens 116 b and been reflected from atriangular or pyramidal-shaped mirror 114. The light emitted fromscanning optical fiber 72/72′ is also focused by lenses 112 a and 112 b,as well as one of lenses 116 a and 116 b (depending upon which side oftriangular-shaped mirror 114 the light is incident and reflected). If apyramidal-shaped mirror 114 is employed, lenses (not shown) could beincluded on the side facing into the drawing and on the side facing upin the drawing, to collect light from those two side, as well as fromthe two sides that are shown.

While not required on the flexible endoscope, in many applications, itwill be desirable to take a biopsy sample from a nodule or other suspecttissue adjacent to the distal end of the flexible endoscope. Whilevarious techniques can be employed to take the biopsy, FIGS. 1F, 1G, and1H illustrate an exemplary embodiment of a flexible endoscope 24″ thatincludes a rigid portion 26″ with a biopsy lumen 117 in which isdisposed a flexible shaft 119 having a biopsy needle 121 disposed at itsdistal end. Also, by detecting light using photodiode sensors at thedistal end, as shown in FIGS. 1C and 1D, there is sufficient roomproximal to the scanning fiber to encompass both the biopsy channel andtracking sensor 84′ within a 2 mm diameter housing. The additionaloptical sensor wires can be combined with fine tube piezo actuator wires123. When the flexible endoscope has been positioned and oriented (asdetermined by the present invention) so that the distal end of thebiopsy needle is disposed adjacent to a tissue mass that is to bebiopsied, flexible shaft is advanced distally, forcing biopsy needle 121to extend outwardly of biopsy lumen 117 and into the tissue mass. A plugof the tissue mass remains embedded in the biopsy needle and it can bewithdrawn from the site of the biopsy. The specimen retained in thebiopsy can then be retrieved when the flexible endoscope is withdrawnfrom the passages.

FIG. 1I illustrates a flexible endoscope 24′″ in accord with the presentinvention, that includes a cytological brush 125 for taking tissuesamples from the walls of the passage through which the flexibleendoscope is advanced. This brush can be advanced over the largerdiameter portion of the distal end of the flexible endoscope to take asample of tissue cells at a desired location using a tubular shaft (notshown) to hold the cannula-style cytological brush. For purposes ofsimplification, this Figure does not illustrate all of the othercomponents of the flexible endoscope, but it will be understood thateither a position sensor or an electromagnetic transmitter will beincluded in a rigid portion 26′″ adjacent to the distal end of theflexible endoscope, generally as described above. In addition, thisembodiment will also include either light collecting optical fibers orlight detectors to provide an image of the adjacent region. As is trueof the other preferred embodiments of the flexible endoscope, in thisembodiment, scanning optical fiber 72 is driven so that it scans in aspiral pattern 127.

Model Generation and Registration

While not required, it is preferable, when navigating through a complexset of linked passages in the body, to employ a model of the linkedpassages so that a route toward a desired point in the body that is inor adjacent to the linked passages. A graphical surface model of thepatients bronchial tree can be provided or can be generated from HRCT orMRI scans using a well-known method for segmentation, such as local 3-Dregion growing or more complex algorithms that incorporate morphologicalapproaches (e.g., as taught by Kiraly et al., in “3-D human airwaysegmentation for virtual bronchoscopy,” Proc of SPIE 4683 Med Imag.,2002). Next, an airway tree model is created from the segmented volumeusing 3-D skeletonization and branch point detection techniques (e.g.,as taught by Kiraly et al., in “Three-dimensional path planning forvirtual bronchoscopy,” IEEE Trans. Med. Imag., 23(9), 2004). FIG. 3Aillustrates a plurality of HRCT scans 162 that have been made of apatient 160 and used to produce a 3-D airway tree model 164. Suchtechniques have been demonstrated in virtual bronchoscopies. The lateralresolution of HRCT is on the order of 1 mm, (slightly smaller than thesmallest airways accessible to the flexible endoscope).

As shown in FIG. 2A, branch points 128 in a model 120 are identified asthe intersection of all airways whose position is defined as the innerpoint of confluence of the two subtending vessels. For example, as shownin FIG. 2C, an airway 140 branches into two airways 142 and 144. Anarrow 146 points toward the main carina of the bronchial system, whilearrows 148 and 150 point away and down two subtending passages 142 and144, respectively. The bold arrow points to the branch point of thesethree airways. They can be found automatically by skeletonizing surfacemodel 120 to form a series of centerlines 126 that comprise the airwaytree model and selecting all points where these centerlines intersect asbranch points 128, as shown in FIG. 2A. A 3-D Cartesian coordinatesystem 122 is preferably associated with the 3-D model and can have anorigin 124 disposed at the first branch point of the bronchial tree,which is the main carina. Because a branch point 155 in the lung tree isfound from the intersection of centerlines 154 through skeletonization,this point may not correspond to the actual inner point of confluence,as shown in FIG. 2-D. The appropriate branch point can be defined byprojecting a line 158 from this intersection at an angle half that of abranching arc 156 formed by two subtending vessel centerlines 152 and153, until it intersects with the surface model branch point.

The generation of each node in the skeleton model is determined bycounting the number of connections separating it from the main carina.From the example shown in FIG. 2B, it will be evident how a hierarchaltree is computed where every branching node is represented by itsgeneration (e.g., second generation branches 130 that are separated fromthird generation branches 132 by branch points 136, and so forth,through fourth generation branches 134) and its x-y-z position withinthe model coordinate frame (MCF). In this example, the MCF is thecoordinate system of the lung model. The origin of this coordinate planemay be chosen arbitrarily. For example, the main carina may be chosen asa reference point or origin at coordinates (0,0,0) as shown in FIG. 2A.This coordinate system is otherwise arbitrary and is not affiliated withthe position-sensor coordinate system initially.

Once the origin is specified relative to some anatomical feature orother arbitrary point, the MCF is confined to the coordinate space inwhich the physician maneuvers the device in real-time, otherwisereferred to here as the absolute coordinate frame (ACF). While simplepoint to point alignment does not fully constrain the 3-D lung model inthe model coordinate frame (MCF) into the ACF, one or more externalposition sensor(s) can be affixed to external point(s) on the patient tomonitor changes in the orientation of the chest cavity over time due topatient breathing or shifting, as discussed in greater detail below inconnection with FIG. 7C.

In one embodiment, the ACF is established by a stationaryelectromagnetic field transmitter from which the position sensordisposed adjacent to the distal end of the flexible endoscope determinesits position and orientation. The system allows for calibration at anyarbitrary point 85 in the ACF. When calibrated, this point becomes theorigin or reference point of Cartesian coordinate system 122 in the bodyof patient 160, for both the ACF and MCF, as shown in FIG. 3B. In orderto track the sensor position within the MCF, the two must be registeredsuch that a known position in the MCF correlates to a known position inthe ACF. A variety of points can be chosen for this purpose, such as themain carina of the patient (as noted above), the collar bone, or anyrigid structure identified on the model and accessible to the clinician.It is assumed at this stage that the relative size and shape of thebronchial tree has not changed significantly from the time of the CTscan that was used to produce the 3-D model. A variety of non-rigidregistration techniques can also be used by placing additional sensorson the patient and modeling the deformation of the lung due tophysiological functions. Because this approach requires registering adiscrete number of points in 3-D space, a simpler non-rigid registrationmethod, such as a linear elastic modeling of branch point deformationconnected by links would be preferable. However, if the deformation dueto tidal breathing can be measured empirically, then the model can beregistered from observed data rather than using a heuristic model ofairway deformation.

Selecting Points of Biopsy and Course Plotting

Within the MCF, region(s) of interest are selected as intended points ofbiopsy. Typically, these are nodules or masses found on HRCT scans, andthus have a predefined position in the MCF. The surrounding airways canbe elected for interrogation automatically or chosen by the clinician.As shown in FIG. 4A, a series of paths 172 and 174 through bronchialpassages 170 are generated based on the branching nodes 176 leadingthrough these airways, for example, to a point of biopsy 178, which isadjacent to a nodule 180. The layout of this course is analogous to aroadmap where navigation relies on a series of decisions or turns onewould make in route to reach a desired destination.

Implementation of a Graphical User Interface

Contained within the user interface are windows displaying the 3-D lungsurface model and pre-procedural path planning (FIG. 4A), the real-timevideo image (FIG. 4B), and a virtual fly-through perspective constructedfrom the CT scan (FIG. 4C). A graphic marker 192 is displayed in theuser interface of FIG. 4B to show the position of the catheter inairways 190, and the intended navigation routes to the points of biopsyare shown in FIG. 4D.

Guided Navigation—Position Calibration

As the scope traverses the airways, the graphical interface iscontinually updated, charting progress from both global and fly-throughperspectives. With reference to FIG. 5A, as the scope approaches abifurcation 210 in a bronchial passage 212, the user interface indicateswhich subtending vessel to enter. However, it is necessary to ensurethat the scope end is actually at a bifurcation (i.e., that the MCF andACF are still registered correctly). Before steering the flexibleendoscope down either branch, the clinician verifies the position of theflexible endoscope using a touch sensor on the distal end of theflexible endoscope or by visual assessment if necessary. The distal endof the flexible endoscope is touched to the inner point of confluence ofthe two subtending vessels (i.e., to bifurcation 210) and an input fromthe clinician on the controller processor initiates a recalibration ofthe two coordinate frames by registering a position 214 in the MCF ofthe model to a current actual position 216 in the ACF measured by theposition sensor. Model updating can be initiated by the clinician usinga foot pedal (not shown) that signals the computer to recalibrate thevirtual model by re-registering.

Roadmap Decision Model

Based on the chosen route and the known position and orientation of thecatheter tip, a visual graphic is presented on a video monitor toinstruct the clinician on how to proceed.

There are three methods described here for assisted navigation at eachnode in a series of linked bronchial passages: (1) the entire course isplotted within the virtual fly-through and global model perspectives andthe clinician must resolve which path to take in the endoscopic imagingwindow through direct comparison; (2) given the calculated branchingangle subtended by the daughter vessels in the airway tree model and aswell, the viewing angle of the scanning fiber endoscope, a visual markeris overlaid on top of the endoscopic video images to indicate theintended direction of travel; and (3) image analysis in a bronchialpassage 230 is used to segment regions of low intensity by thresholdingimage and selecting candidate airway regions 238 and 240, which areassociated respectively with passages 232 and 234, and a specific pathis chosen based on how closely it correlates with the airway tree modelstructure referenced to Cartesian coordinates 236 (FIG. 5C). Again, asshown in FIG. 5B, a visual marker 222 (e.g., green) is overlaid on topof the scope images, explicitly indicating a correct path choice 220.Here, the deflection angle is represented as a vector of two elementswhere there is a component of deflection in both the x and y axis of the2-D endoscopic image. An additional indicator 224 (e.g., red) is alsoused to signal that a particular region has already been inspected.

Shortest Path Navigation

In many cases, the desired biopsy point lies beyond the airway treemodel. It is difficult to segment small airspaces on HRCT and MRI scansand often the reconstructed model is missing branches of the bronchialtree resulting in an unresolved termination of several vessels. However,with reference to an exemplary illustration 250 shown in FIG. 5D,another navigation method can be used, with a dual purpose of navigatingthe physician to the correct location adjacent nodule 180, as well asextending a model 252 by “growing” the airways, as the flexibleendoscope is advanced beyond the model bounds. This technique, asopposed to a roadmap method, operates by minimizing the distance betweenthe present position and the intended destination at every step alongthe way. This straight shot approach makes the assumption that theseemingly more direct route is the correct route. Obviously, this is notnecessarily true and some intervention must be made to correct for wrongdecisions. As the remaining airways are traversed, some threshold willbe used to signal that the chosen course has deviated to far from thechosen point of biopsy. A record of trial and error navigation isconstructed that keeps track of what vessels did not lead to theintended destination. In essence, this decision model operates similarlyto a “hot and cold” scenario where a predefined threshold for “cold”governs how far the scope is allowed to veer from the known position ofthe mass as a direct Euclidean distance, i.e., by setting a limit forthe amount of deviation from the desired location toward which theflexible endoscope is being advanced before the current path must bereversed to at least a last branch point. This embodiment of thenavigation technique may be used exclusively and may be more practicalin other applications where there is little branching.

Model Extension

When the physician approaches a branch point that is beyond the limit ofthe model, rather than re-registering, the airway tree model can beextended at desired points to establish decision points along the way,while navigating to the point of biopsy. FIG. 5D illustrates how thenavigation history is updated and added to airway tree model 254,extending from the open termination of the model of the airways. Themodel is extended by using the flexible endoscope itself as a tool formapping rather than from segmented images. Under this scenario, theclinician prompts the computer using some mechanism (e.g., a controlbutton, a foot pedal (neither shown), or a voice command) to recordposition and orientation measurements and add them to the airway treemodel, along with archiving the endoscopic image to the library ofimages associated with that specific location in the airway tree model.Similarly, erroneous paths or travel markers can be deleted if toosusceptible to measurement error.

It is also possible to gate the position measurement at an inspirationlevel that is the same as that when the CT or MRI scan was taken. Thisstep is achieved by placing one or more external sensors such asposition sensor(s), impedance electrode(s), or strain gauge(s) on thesternum, or other such sensor on the thorax, as shown in FIG. 7C, whichwill then serve as a surrogate for measuring lung volume and thus, lungdeformation. This step is done under the assumption that the lungdeformation is directly correlated with respiration. Similarly, if usinga dynamic lung model in which the deformation of branch points aredirectly measured, several position readings should be taken in order toaccurately predict deformation for dynamic non-rigid registration.

Navigation History

When obtaining tissue specimens from a plurality of locations throughoutthe lung, it becomes necessary to track the catheter in order todetermine a travel history that is displayed on the lung model tree. Bydoing this, it is possible for the clinician to keep track of lobes thathave been explored as well as to revisit a certain region during afollow-up examination. This tool greatly assists the doctor in assuringthat all accessible lung regions adjacent to an identified mass havebeen explored. Additionally, the time expense associated with aphysicians need to continually re-orient the catheter to some knownposition will be greatly diminished while relieving them of the burdenof mentally memorizing all explored and unexplored lung regions. Thisprocedure can be done easily by “tagging” branches that have beeninspected on the graphical interface; FIG. 5B illustrates a color-codingscheme for labeling explored and unexplored regions. A comparison can bemade with a prior evaluation by loading results of the biopsy pertinentto the specific area.

Imaging Techniques for Motion Detection

Despite a physician's best efforts to correctly recalibrate the systemto accurately define the tip position within the MCF, a large amount oferror impedes the ability to navigate past a certain point, especiallywhen beyond the limits of the 3-D model. In this respect, a subsystem isdescribed for precise navigation along many of the more tightlybranching bronchioles of the peripheral lung. Image analysis ofsubsequent frames obtained from video can be used to analyze movement. Adifference image is defined as the subtraction of two images and can beused to detect movement when a given region demonstrates a high degreeof change in the pixel values above what is typically observed fromnoise. In this scenario, difference images are used to discriminatemotion induced by noise, error, or breathing (no appearance of motionfrom the scope's perspective) versus motion resulting from actualextension of the catheter along the vessel path. Once the scope ismoved, the position tracking initiates again and does not stop untilmotion has desisted, as determined from the difference images obtainedfrom the video.

Tracking Position/Orientation of Flexible Endoscope

In one embodiment shown in FIG. 7A, the actual position and orientationof the distal end of flexible endoscope is tracked using an externalelectromagnetic field transmitter 276 to produce an electromagneticfield to which a sensor 278 responds by producing corresponding signalsindicative of the position and orientation of the distal end of theflexible endoscope. These signals, which are conveyed along with lightsignals through a lead 280 that is coupled to the proximal end of theflexible endoscope, are processed by external components (i.e.,components in a box 282). The flexible endoscope is inserted into thebronchial passages of a patient 272, who is reclining on a non-ferrousbed 274. The electromagnetic field readily penetrates the body of thepatient, enabling the realtime tracking of the flexible endoscope in theACF as the flexible endoscope is advanced through the passages in thebronchial system.

In an alternative embodiment shown in FIG. 7B, an internalelectromagnetic field transmitter 290 can be affixed adjacent to thedistal end of the flexible endoscope, and one or more external sensors292 can be employed to respond to the electromagnetic field produced bythe internal electromagnetic transmitter, providing correspondingsignals that are processed by light source and processor 282 to againdetermine the position and orientation of the distal end of the flexibleendoscope. It is also contemplated that other forms of transmitters andsensors might instead be employed to monitor the position andorientation of the distal end of the flexible endoscope. For example, anexternal transmitter emitting modulated infrared (IR) light might beemployed with a corresponding IR sensor that responds to the IR lightreceived as the light passes through the body of the patient.

Details of an exemplary dipole electromagnetic field transmitter 400 anda corresponding sensor 402 are illustrated in FIG. 9A. As shown in thisFigure, electromagnetic field transmitter 400 includes threeorthogonally disposed coils 404 a, 404 b, and 404 c, while sensor 402includes three orthogonal coils 406 a, 406 b, and 406 c. While neitherthe coils in the transmitter nor the coils in the sensor must beorthogonal to each other, it is preferable that they be so, since theprocessing of the signals produced by the coils in the sensor is therebysimplified, and the results are likely to be more accurate. As notedabove, the electromagnetic field transmitter may be either external tothe patient's body or internal, with the sensor being correspondinglyinternal or external to the patient's body. Appropriate sizing of thetransmitter or the sensor is required to enable that component to befitted adjacent to the distal end of the flexible endoscope.

An alternative configuration of a sensor 402′ that includes only twocoils 406 a and 406 b is illustrated in FIG. 9B. If only five degrees offreedom (DOF) are required, then a sensor 402″ having only one coil 406b can be employed, as shown in FIG. 9C. It may be preferable to includemore coils in either the transmitter and/or receiver than required,since the redundant coil(s) ensure(s) that the position and orientationof the distal end of the flexible endoscope is accurately determined.The product of the number of coils on the sensor and the number of coilson the transmitter must at least equal the number of DOF to bemonitored, with the caveat that at least two coils are required on thesensor to sense six DOF. With the ability to determine only five DOF inthe embodiment of FIG. 9C, the rotational orientation of the sensor (andthe flexible endoscope) will typically be excluded from thedetermination; however, rotational orientation may not be important insome applications of the flexible endoscope.

Use of Bodily Function Sensor to Minimize Error in Tracking

The embodiment of FIG. 7C includes an optional band 294 fastened aroundthe thorax of patient 272, to mount a bodily function sensor 296, whichresponds to the movement of the patient's chest during breathing, toproduce a corresponding signal indicative of this bodily function thatis conveyed through a cable 298 to light source and processor 282. Thissignal can be used to synchronize measurements of the position of thedistal end of the flexible endoscope with the inhalation/exhalationcycle of the patient, to minimize errors caused by the displacement ofthe flexible endoscope within the bronchial passages as a result ofrespiration and movement of the bronchial passages with respiration, orto match the respiration levels of the patient to those at the time theCT or MRI scans were made. FIG. 6 shows how the position of the flexibleendoscope can be corrected for the effects of breathing. However, it maybe preferable to only take measurements of the position of the flexibleendoscope when the patient's breathing cycles are not moving air throughthe bronchial passages, i.e., at a held-lung volume. The signal producedby sensor 296 enables this option to be readily effected.

FIG. 7D illustrates yet another approach in which an imaging device 299,such as a laser scanner or a video camera, is employed to scan a patternof reflective dots 301 that is applied to the thorax of the patient. Theimaging of these dots in realtime can produce a graphics signal thatindicates the movement of the chest, either due to patient motion or asa result of breathing. The number of sensors used depends on the numberof measurements required to determine lung deformation given an a priorimodel of the bronchial tree dynamics or an empirical model linkingbronchial tree deformation with lung volume.

As shown in FIG. 7E, it is also contemplated that one or more othersensors 303 comprising, for example, position sensors, impedanceelectrodes, or strain gauges, can be affixed to the thorax of thepatient to produce signals directly indicative of lung volume ordeformation of the bronchial passages. The one or more motion sensorsare coupled to a movement interface 305, which preprocesses the movementsignals and supplies a processed movement signal to the processor inlight source and processor 282. Alternatively, it may be preferable toemploy a signal produced by one or more sensors that is a surrogate fordetermining the movement of the bronchial passages. For example, if theexternal signal is monitored while the flexible endoscope is at a knownposition, any change in the position of the flexible endoscope duringbreathing or movement of the chest can be empirically related to theexternal signal produced by the one or more sensors. Thereafter, theexternal signal from the one or more sensors can be used indirectly todetermine the movement or deformation of the bronchial passages. Adynamic model that accounts for tidal breathing may be obtained from CTscans at multiple levels of inspiration, from position sensing at eachbifurcation during the bronchoscopy, or from position sensing performedduring a previous bronchoscopy.

Logical Steps Employed in Tracking Flexible Endoscope

FIG. 8A illustrates a flow chart 300 showing the logical steps that arecarried out in the present invention. These steps include carrying out aCT scan (or other imaging modality) in a step 302, performing avolumetric segmentation of the bronchial tree, as indicated in a step304. In addition, the imaging of the lung can be employed to identifyany nodules or other regions of interest, where a diagnostic proceduresuch as a biopsy is to be performed, as indicated in a step 306. In astep 308, the clinician conducts pre-procedural planning, e.g., todetermine the steps that should be taken to reach a desired region andcarry out the diagnostic procedure, the type of biopsy that will betaken, and other such details. A computer is used to implementskeletonization and branch point detection of the segmented image data,in a step 310, producing results such as shown in FIG. 2B. In a step312, a six or five DOF tracker is readied to enable the flexibleendoscope position and orientation to be tracked as the distal end offlexible endoscope is advanced through the bronchial passages.

The flexible endoscope will produce a plurality of endoscopic images 313as it is advanced through the bronchial passages. An array of suchimages 315 that include overlapping portions can be employed to produceone or more stitched images 342 that are texture-mapped to the surfacemodel. Adjacent images are combined by subtracting one image(pixel-by-pixel) from another image and iteratively moving one image inspace to determine the optimal region of overlap, which corresponds tothe areas where the error between the two images is minimized. Thistechnique is well-known in the digital image art, and many digitalcameras that are currently sold include software for producing apanoramic image (which is completely analogous to the stitched image)from two or more adjacent images. Alternatively, rather than subtractingcomplete images, matching regions of interest can be identified in bothimages using a fast search algorithm and then computing multiple motionvectors that provide a good estimation of the total displacement.

Within a virtual interface provided by the model, a step 314, providesfor making an initial registration between the MCF and the ACF, asdiscussed above. This step ensures that some designated point (such asthe carina) that is apparent in the model frame of reference isinitially set to correspond to the actual location of the flexibleendoscope determined by the tracking system. As discussed above, threedifferent approaches can be employed for adaptively updating andre-registering the MCF to the ACF to reach the nodule(s) or otherregion(s) of interest. These methods include the shortest (or mostdirect) path approach in a step 316 that is used when moving beyond theextents of the 3-D model, to produce a model extension 328. Within themodel, an adapted rigid registration approach can be used, as indicatedin a step 318, or an adaptive non-rigid registration approach, asindicated in a step 320. These two approaches can employ automaticcalibration using image-based tip tracking (at bifurcations of thepassage) at a step 322, manual calibration at automatically selectedpoints in a step 324, or manual calibration at arbitrarily selectedpoints in a step 326, to register the MCF and ACF. At each of thesebranch points, endoscopic images 313 can be stored in an array ofendoscopic images 315 for annotation or later inspection and saved in alibrary of images 336 within the lung model, for texture mapping 343 andimage stitching 342.

Within the ACF, in a step 330, the registration can be carried out in asingle measurement at a respiratory-gated level (e.g., determined by oneof the bodily function sensors described above), or in a step 332, asmultiple measurements of position over the respiratory cycle. The resultof either of these two steps is a back registration to the MCF, asindicated in a step 334 for the purpose of establishing a navigationhistory and charting the clinician's progress in moving the flexibleendoscope through an annotated airway tree model.

One goal of the present invention when used in bronchial passages is toprovide an annotated lung tree to provide a data record of each of thebronchial passages that was traversed both in regard to the 3-D model,and in regard to mapping the passages and the actual position andorientation of the distal end of the flexible endoscope as it wasadvanced through these passages. This record will also includeendoscopic image library 336 of selected images produced by the scanningoptical fiber as it passes through the bronchial passages. The initial3-D model corresponds to a static airway tree model 338 that is used todisplay position, record and present findings, and fuse 2-D endoscopicimages to a 3-D surface. This model is the basis for a dynamic airwaytree model 340 that is more accurate and compensates for the effects ofbodily functions, such as breathing. The images produced by the scanningoptical fiber in the flexible endoscope correspond to a surface model inwhich stitched images 342 are texture-mapped to the inside surface. Forpurposes of possibly revisiting points in a later follow-up procedureand confirming where tissue samples were taken, the surface model willalso include a map of all biopsy points 344. Also important are the datacomprising a navigation history 346 that indicates each passage throughwhich the flexible endoscope passed during the procedure, as well asother data of interest.

FIG. 8B includes a diagram 331 illustrating the alternative steps foroptimized in vivo data analysis that can be employed when updating the3-D model in the MCF, in a step 353, using the position and orientationmeasurements made in the ACF (as noted in a block 333). In a step 348,time-averaging of the position measurement may help reduce error in theposition measurement as it relates to either re-registering the model orextending the model. A step 350 indicates that the position data for thedistal end of the flexible endoscope can be more closely related to theMCF by snapping the position measured to the centerline of the passagein the 3-D model for both real-time virtual image display and as wellfor precisely determining navigation history relative to the centerlinesthat comprise the airway tree model. A step 352 may also be implementedto limit position measurement error due to breathing, by only trackingthe motion of the endoscope along its axial direction. This movement maybe determined as the dot product of the calculated motion vectorextending between two points and the forward vector defined by thedirection in which the distal tip is pointing. The image shown in FIG. 6illustrates the view of a small bronchial 161 at inhalation andexhalation. The overall motion of the catheter is measured as thecomponent of an overall motion vector 163 along a forward vector 165 toobtain an error-corrected motion vector 166.

Details of step 316 for determining the shortest or most direct pathwhen navigating beyond the limits of the 3-D model are shown in FIG. 8C.As part of the model extension, it may be necessary to make severalposition measurements over time to generate either a probabilistic modelindicating where small bifurcations are located, or to assist indeveloping a empirical model for airway deformation as a function ofrespiration. At a bifurcation, in a step 354, a clinician selectivelydetermines the position and orientation of the distal end of theflexible endoscope based upon the sensor readings. If registration gatedposition sensing is used, a single measurement of position andorientation is taken in a step 356. If dynamic position sensing is beingused, a step 358 provides for taking multiple measurements over anentire respiration cycle. Based on the single or the multiplemeasurements, in a step 360, the coordinates in the ACF areback-registered to the MCF (i.e., to the last position in the model). Astep 362 then provides for adding the new coordinates to the lung tree,so that the model is extended to the current position. It is preferable,that from the current position, a step 364 will then provide an arrow onthe virtual interface that indicates a direct path and the distance tothe intended destination. While not shown here, if the distance from theintended destination should continue to increase beyond an acceptedlimit, the clinician may be directed to back up to a previous one ormore bifurcation and take a different path.

In FIG. 8D, details of step 322, which provides for automaticcalibration using image tracking are shown. A step 366 provides forregistering the MCF with the ACF at the current position and orientationof the distal end of the flexible endoscope. Three events can be used toinitiate the automatic calibration or registration. In a step 368, theregistration procedure is initiated when motion of the distal end of theflexible endoscope is detected by the sensor. In a step 369, motion isdetected when the difference image computed by subtracting an acquiredendoscopic image and the endoscopic image stored during the lastreregistration exceed some threshold. Alternatively, in a step 370, theclinician or other user may initiate the registration by some input. Astep 372 indicates that an initial estimate of the new position andorientation is determined based upon the change in position andorientation of the sensor from the previous position and orientationmeasured during the previous reregistration. During a baselinebronchoscopy, a step 376 generates a surface rendered virtual image fromthe 3-D model data. Next, using the actual image data produced by thescanning optical fiber, a step 378 computes a similarity between therendered virtual image and the endoscopic image. As an alternativeoption, during a follow-up bronchoscopy, a step 374 computes thesimilarity between the present endoscopic image and a library ofendoscopic images acquired during the baseline bronchoscopy with a knownposition in the MCF. A step 380 then iteratively increments ordecrements the position and orientation parameters independently in theMCF, thereby generating a new hi-lighted surface model image or newperspective image of the texture-mapped surface model and computing anew image similarity. If the image similarity has increased, the processrepeats by looping back to step 376. However, if the image similarityhas not decreased, a step 382 re-registers the MCF to the ACF at thelocal maximum (i.e., maximum similarity).

There are several ways to determine similarity. Information for thevirtual image is not in the structure of the 3-D model, but is based ona projection of the surface rendered model to a two-dimensional (2-D)image, which is done by assuming a point source of white light at thedistal end of the flexible endoscope. The virtual model is then renderedas a lighted surface at an appropriate intensity, cone angle, andattenuation factor. These parameters are manipulated to yield the bestresult (the result that best mimics actual tissue and the flexibleendoscope design). Color is not important; the process involvescomparing image brightness between endoscopic and virtual images (astaught by Mori et al. in “A method for tracking camera motion of realendoscope by using virtual endoscopy system,” Proc. of SPIE 3978, Med.Imag., 2000). Image similarity is then computed as follows. An imageerror E is calculated by determining the means squared error (MSE) ofthe two images and dividing by the correlation (Corr) of the images:E=MSE(I _(virtual) ,I _(endoscopic))/Corr(I _(virtual) ,I _(endoscopic))Minimizing E will produce the most similar match between images.

FIG. 8E includes details of step 320 for adaptive non-rigidregistration, for both manual calibration, and for automaticcalibration. Under manual calibration, a step 384 determines the ACFposition of the distal end of the flexible endoscope, while under theautomatic calibration, a step 386 provides that the MCF is automaticallyre-registered to the ACF by image-based tip tracking at bifurcations inthe passage. Two alternatives can then be followed for either manual orautomatic at this point. If an a priori model of the bronchial tree orpassage deformation based on measuring the position of external sensoris employed, a step 388 measures the position of the flexible endoscopeusing multiple sensors or using a plurality of markers that have beenplace on the thorax of the patient (as discussed above in connectionwith FIGS. 7D and 7E). If an empirical model of the bronchial treedeformation is used instead, a step 390 provides for measuring orestimating lung volume (which is only a surrogate for measuringdeformation of the bronchial passages). Lung deformation can be foundempirically either by taking multiple CT scans at different breathlevels, measuring the change in position of the flexible endoscope ateach bifurcation using the position sensor, or by using anotherheuristic model that links respiration level and airway distension. Themeasurement of respiration level can be taken from the one or moresensors placed on the chest, as in FIG. 7C.

After either step 388 or step 390, a step 392 non-rigidly registers theMCF to the ACF. A step 394 hi-lights a bronchial passageway that leadsto an intended destination, on the virtual image provided on thedisplay, so that the clinician knows which branch to follow whenadvancing the flexible endoscope.

In FIG. 8F, details of step 318 for adaptive rigid registration areillustrated. Again, the registration can be done manually, orautomatically. Under manual registration, two approaches can be used. Ina step 395, the ACF position is measured using the sensor disposedadjacent to the distal end of the flexible endoscope. In connection withthis step, in a step 396, the position is measured using a plurality ofsensors that are externally placed on the patient's thorax (i.e., asshown in FIG. 7E). After step 396, a step 397 provides that the MCF isrigidly registered to the ACF.

If automatic registration is used, a step 398 provides that the MCF isautomatically re-registered to the ACF by using image-based tip trackingat a bifurcation of the passage. After either step 397 or step 398, astep 399 provides that the bronchial passageway leading to the intendeddestination is hi-lighted on the virtual image so that the clinician canknow the appropriate branch to follow.

Other Considerations

This invention embodies any means of navigating a branching system invivo including the cardiovascular system, lymphatic system, nasalpassages, secretory glands and organs, urethra, pancreas and kidney.Considerations to the proposed methodology may include: (1) other formsof obtaining 3-D models such as using MRI or multimode images such asPET (would it be useful for delivering nuclear contrast agents directlyto the lung mass?) and CT (2) determination of catheter position usingfluoroscopy or other means of external tracking; (3) path finding basedon other deterministic parameters such as deflection angle and insertionlength. Another metric for determining position relies on measurementsmade of the vessel diameter made by side viewing embodiments of theproposed flexible endoscope, such as shown in FIG. 1E, or by using anultrasound transducer (not shown) mounted at the distal end of theflexible endoscope.

Alterations to the currently proposed method might utilize side-viewsfrom a flexible endoscope to locate branch points. Additionally, othervisual cues may be applied to this methodology that do not includebranch points. In navigation through other branching systems,identification of organs, known abnormalities such as tumors or lesions,or fiducial markers including biocompatible dyes. Discriminatingposition may also be accomplished by correlating current images obtainedfrom video frames with images already obtained during the procedureusing auto-correlation.

In addition to the number of error correction strategies mentioned here,it is also possible to model the lung as a deformable structure anddetermine probabilistic position of the scope based on establishedmeasures of position error obtained from experimentation. Luckily, theflexible endoscope serves as an ideal tool for measuring the extent ofdeformation or inflation of the lung tissue. Similarly, the lung willdeform due to shear stress applied by the flexible endoscope itselfparticularly within the smaller vessels. Prior knowledge of the tissuecharacteristics within both the central and peripheral lung help toestimate error in the position measurement. Given this, a form of hapticfeedback might be desirable to measure direct forces imposed on the lungby the bronchoscope. In the event that too many errors have accumulateda complete reinsertion and calibration of the ACF and MCF may bedesirable.

With regard to the user-interface itself, the 3-D model should be ableto be rotated in all directions. Annotations could be logged in atextual or audible form for comparison at a later date. Regions thatcould not be accessed in previous examinations would have some visualindicator that links to a set of comments or notes made by the same ordifferent physician at an earlier date. Controls to the user-interfacecould potentially be run by voice command given the preoccupation of thedoctor with the manipulation of the bronchoscope. Also, 3-D modelviewing could be improved by using an autostereoscopic display orstereoscopic head-mounted display.

Detailed description of the exact route used for navigating to multiplebiopsy sites could be integrated into robotic surgery applications sothat bronchoscopies could be performed from a remote location whilecontinuing to monitor patient health real-time.

While ultrasound is not commonly used for imaging of the lung due to thehigh impedance of air, measurements made at the periphery may providereasonable estimates of the scopes position in vivo.

Although the present invention has been described in connection with thepreferred form of practicing it and modifications thereto, those ofordinary skill in the art will understand that many other modificationscan be made to the present invention within the scope of the claims thatfollow. Accordingly, it is not intended that the scope of the inventionin any way be limited by the above description, but instead bedetermined entirely by reference to the claims that follow.

What is claimed is:
 1. A system for visually guiding a flexibleendoscope through linked passages within a body, comprising: a flexibleendoscope; a sensor that produces an output signal indicatingcoordinates that define a three-dimensional disposition of a distal endof the flexible endoscope; a processor configured to generate athree-dimensional model of the linked passages, determine positions ofbranching points of the three-dimensional model of the linked passages,and provide indicators that indicate the positions of the branchingpoints on the three-dimensional model based on the coordinates indicatedby the output signal in response to the flexible endoscope beingadvanced beyond a limit of the three-dimensional model; and a displayfor displaying a current view from within the linked passages generatedby the flexible endoscope as the flexible endoscope is advancedtherethrough, to enable the flexible endoscope to be visually guided andtracked along a path through the linked passages, wherein the outputsignal indicates coordinates that define a disposition of the distal endof the flexible endoscope relative to the three-dimensional model as theflexible endoscope is advanced and wherein the display shows a currentposition of the flexible endoscope relative to the three-dimensionalmodel of the linked passages.
 2. The system of claim 1, wherein theprocessor and display are configured for a user to identify bifurcationsof the three-dimensional model based on the current view from within thelinked passages in order to confirm alignment of the disposition of thedistal end with the three-dimensional model.
 3. The system of claim 1,wherein the processor and the display are configured to provide saidthree-dimensional model on said display to enable determining at leastan initial path along which to advance the flexible endoscope throughthe three-dimensional model of the linked passages.
 4. The system ofclaim 1, wherein the three-dimensional model is derived from an image ofa portion of a body in which the linked passages are disposed, thethree-dimensional model providing an indication of an orientation and adisposition of the linked passages relative to a specific anatomicalfeature.
 5. The system of claim 1, further comprising a signal source,the signal source comprising an electromagnetic field source, andwherein the signal source and sensor together comprise at least fivecoils configured to sense at least a three-dimensional position of adistal portion of the flexible endoscope relative to the electromagneticfield source.
 6. The system of claim 5, wherein the sensor comprises atleast one coil configured to sense both a three-dimensional position andan orientation of the distal portion of the flexible endoscope.
 7. Thesystem of claim 1, further comprising a body function sensor configuredto be disposed adjacent to one of a sternum, a thorax, or a mouth of apatient, to monitor at least one of breathing, a chest orientation, or adeformation of airways of a patient, for at least one of updating thethree-dimensional model in regard to changes caused by at least one bodyfunction, or gating a determination of a disposition of the distal endof the flexible endoscope in regard to the at least one body function.8. The system of claim 1, wherein the flexible endoscope includes avibrating optical fiber that scans a region adjacent to the distal endof the flexible endoscope, thereby producing a signal that is used todrive the display.
 9. The system of claim 1, wherein the flexibleendoscope is sized to be advanced through an 8th generation bronchiolein a patient's lung.